1. Field of the Disclosure
The present disclosure relates to the field of mobile video processing, and more particularly relates to a panoramic video processing method and device as well as a non-transitory computer-readable medium.
2. Description of the Related Art
With the rapid development of photography apparatuses and mobile networks, hand-held camera based video capture has been widely applied in people's lives, for example, mobile live video platforms, live travel, and live shopping. In order to provide good quality videos for users, video stabilization plays a very important role in real time video processing.
At present, there are mainly three approaches able to achieve video stabilization, namely, mechanical processing, optical processing, and digital processing. The mechanical processing based approach is using an additional sensor such as a gyroscope, an angle sensor, or the like to acquire motion information of a camera for carrying out motion compensation. The optical processing based approach is utilizing an optical apparatus to adjust the direction and distance between a lens and a prism so as to remove the shaking of a camera, for instance. These two kinds of approaches are both on the grounds of additional hardware, so it is usually necessary to adopt expensive devices, and their portability is not good. The approach based on digital processing is obtaining shaking related motion information by way of image processing, and then, attaining a stabilized image sequence (i.e., a video) by means of image compensation without employing any additional mechanical or optical device. As such, compared to the approaches based on mechanical and optical processing, the digital processing based approach is of wide application.
In addition, on the basis of whether to gain a stable video in real time, the video stabilization approach based on digital processing may be divided into on-line video stabilization and off-line video stabilization. Because the off-line video stabilization approach is a type of video post-processing, the motion trajectory of the whole image sequence may be procured. Thus, it is possible to estimate an optimum smooth path (locus) so as to acquire a more robust and stable video stream. On the contrary, the on-line video stabilization approach cannot take advantage of motion information in the future, so its robustness is not good compared to the off-line video stabilization approach; however, timeliness may be guaranteed. Regardless of the on-line or off-line video stabilization approach, the digital processing based video stabilization approach mainly includes three steps, namely, motion estimation, motion smoothing, and motion compensation. The motion estimation is estimating the initial motion trajectory of a camera. The motion smoothing is calculating a smooth motion trajectory according to the initial motion trajectory. And the motion compensation refers to conducting motion compensation in regard to the initial video on the basis of the smooth motion trajectory so as to gain a stabilized image sequence.
In general, a conventional video stabilization approach adopts a two dimensional (2D) affine transformation model so as to conduct motion estimation based on a rotation matrix and offset matrix, and then, carries out motion compensation along X and Y axes, respectively. As the manufacturing technologies of cameras are developed, 360-degree panoramic cameras are becoming more and more popular. So far, they have been applied to many fields like video surveillance, live streaming, etc. In order to provide better panoramic video quality, panoramic video stabilization techniques are getting people's attention. However, in panoramic video stabilization, due to the particularity of spherical images, motion estimation cannot be conducted by using a conventional 2D motion model. That is, it is impossible to perform video stabilization on a panoramic video in conventional techniques.